I can't believe how efficient you are, Dana, and thank you for continuing this.

One questionable aspect of how we set this up in the flow.cylc was having only one preanalysis script. (We allowed multiple refinediag scripts with the jinja wizardry from @meteorologist15 )

So if the user has multiple preanalysis scripts, we should give the bad news ("Known deficiency that will be fixed in 2026.02").

In our first yaml attempts, we used lists when dictionaries would suffice because that's how the XML looked. But dictionaries are more attractive visually, more pythonic, and supported by uw compose. So that's why I thought to make the preanalysis and refinediag sections dictionaries.

But the dictionary key is the unique user-specified label. It could be vitals but it could be something else. Both of these should be accepted and equivelant.

  preanalysis:
    vitals:
      script: "/home/mdteam/DET/analysis/vitals/gfdlvitals_refineDiag_om5.latest.csh"
      inputs: ['atmos_month', 'aerosol_month']
      do_preanalysis: True

  preanalysis:
    my_db_script:
      script: "/home/mdteam/DET/analysis/vitals/gfdlvitals_refineDiag_om5.latest.csh"
      inputs: ['atmos_month', 'aerosol_month']
      do_preanalysis: True

This kind of validation was something I asked Paul and Christina about for uw tools (because we would be defining arbitrary components for each experiment)!

The verdict: We can have both arbitrary key names (JSON Schema refers to these as "additional properties", i.e. property (key) names that are not explicitly declared) and on top of that, we can still define requirements for their contents.

I haven't explore uw config validate too too deeply, but if this is how they validate, it seems we could definitely use that as well. It seems to relate to our discussions for where the schema could live and how we could potentially define it in the yaml or something. With uw config validate it looks like you pass in both the input path and schema path.

For now (before uw tools work), it looks like we could use additional_properties in the schema and then define the content requirements

I can't believe how efficient you are, Dana, and thank you for continuing this.

One questionable aspect of how we set this up in the flow.cylc was having only one preanalysis script. (We allowed multiple refinediag scripts with the jinja wizardry from @meteorologist15 )

So if the user has multiple preanalysis scripts, we should give the bad news ("Known deficiency that will be fixed in 2026.02").

In our first yaml attempts, we used lists when dictionaries would suffice because that's how the XML looked. But dictionaries are more attractive visually, more pythonic, and supported by uw compose. So that's why I thought to make the preanalysis and refinediag sections dictionaries.

But the dictionary key is the unique user-specified label. It could be vitals but it could be something else. Both of these should be accepted and equivelant.

  preanalysis:
    vitals:
      script: "/home/mdteam/DET/analysis/vitals/gfdlvitals_refineDiag_om5.latest.csh"
      inputs: ['atmos_month', 'aerosol_month']
      do_preanalysis: True

  preanalysis:
    my_db_script:
      script: "/home/mdteam/DET/analysis/vitals/gfdlvitals_refineDiag_om5.latest.csh"
      inputs: ['atmos_month', 'aerosol_month']
      do_preanalysis: True

So with the preanalysis scripts, will it be structured the same as refineDiag here eventually? (with multiple scripts, inputs, ... under different keys) or would it be multiple scripts under "vitals"?

Yes, preanalysis scripts are identical to refinediag scripts except they produce no output. So they will have inputs (which are optional for now), but no outputs.

Yep, looks good!

This example yaml looks good, but it won't actually work because the scripts aren't there yet.

The test cases can come later we agreed.

Right, yeah, just wanted to get the example started